Support incremental builds with probe and enumeration fingerprinting - #162

PathSet holds a list of normalized paths, which cannot express anything beyond
"this file was read". Replacing it with the ObservedPathEntry list is the
structural change the feature needs, and doing it before anything new is
observed keeps it reviewable on its own.

Nothing new is captured here. The sandbox still filters probes and enumerations
out, so every entry produced is a FileContentRead. This schema transition
intentionally invalidates pre-feature cache entries: PathSet changes from
FilesRead to Entries, the feature setting joins the weak fingerprint, and file
reads gain a typed identity in the strong fingerprint. Existing entries degrade
to cache misses and age out through normal eviction.

The rules that only bind once probes and enumerations do arrive come with it,
because they are what the entry list is for and splitting them out would leave a
data structure with no semantics:

* FilterObservations decides what reaches the PathSet. Predicted inputs and
  IgnoredInputPatterns matches are dropped as before, and file reads are still
  gated on the input hasher recognizing the path. Probes and enumerations must
  additionally normalize under the repo or package root: a path under neither is
  not something the cache can reason about, and that is where most of the volume
  and all of the machine-specific noise sits.

* FoldPathSetEntries canonicalizes several observations of one path by the
  precedence order, while keeping enumerations of the same directory under
  different search patterns as separate entries rather than collapsing them.

* The strong fingerprint gains a payload per type. A directory enumeration
  hashes the member list carried on the entry rather than reading the
  filesystem, which is what makes the value reproducible at lookup from the
  cached entry alone.

EnableProbeAndEnumerationFingerprinting arrives off. It contributes to the weak
fingerprint so entries produced with and without it never mix.

PathSet also normalizes a null entry list to empty. It is used as a dictionary
key during lookup, so a payload lacking the property would be hashed -- and
throw -- before any caller could check it. Degrading to a miss is what the null
handling downstream already assumed.

MSBuild grew an EnumeratePattern field on FileAccessData, which
directory-enumeration observations need: without the pattern a filtered
enumeration such as `*.cs` is recorded as if it were unfiltered, so any
unrelated file appearing in the directory invalidates the entry and the common
case essentially never hits. Its absence therefore has to disable the feature
outright rather than degrade it.

MSBuildCache compiles against a reference assembly but runs against whatever
Microsoft.Build.dll the host supplies, so a direct property access would not
compile today and would throw on older hosts if the compile reference were
raised. The property is bound once to a delegate instead, after which each read
costs about a virtual call -- worth caring about, since this sits on the
file-access reporting path.

FileAccessData is a struct. Instance members on a value type take their receiver
by reference, so the delegate does too; the alternative of calling
PropertyInfo.GetValue per access would box the struct every time.

The running MSBuild version is captured alongside it for diagnostics only. The
version that first carries the field is not knowable while it is unreleased, so
probing for the field tests the exact dependency and stays correct if it is ever
serviced back to an older branch.

Nothing consumes this yet.

The sandbox has always reported probes and enumerations; the repository logged
them and threw them away. They are now classified and kept, which is what makes
a cache entry able to depend on a file having been absent or a directory having
held a particular set of names.

Classification is asymmetric on purpose. A probe becomes AbsentPathProbe only
for error codes that mean the path definitively did not exist -- not found, path
not found, bad net path, invalid name -- and everything else, including success,
becomes ExistingProbe. Treating a transient failure such as
ERROR_SHARING_VIOLATION or ERROR_ACCESS_DENIED as absence would let a flaky
machine fingerprint the same sources differently between builds. QuickBuild
classifies the same way.

Enumerate and EnumerationProbe are not the same thing despite the names.
Enumerate is reported against the directory and carries the search pattern.
EnumerationProbe is reported against each matched child -- FindFirstFileEx's
first result and every FindNextFile after it -- so it is an existence probe on
that child. Recording it as an enumeration would key a directory-enumeration
entry on a file path, which lookup-time validation can never re-validate because
it requires the path to still be a directory; every project enumerating this way
would then miss permanently. The directory's own Enumerate observation still
catches member-list changes.

Classification runs before the generic `error != 0` skip, since a probe's whole
value is often that it failed. Accesses carrying Read or Write are content
accesses and keep flowing to the file table untouched.

`*` is normalized to null so an unfiltered enumeration folds to one entry
however the caller spelled it, and the observation list is only allocated when
the feature is enabled, since probes dominate event volume.

Self-outputs are not filtered yet and enumerations carry no member lists, so
enumeration entries currently over-invalidate. The next commit addresses both.
The feature remains off by default.

A project observes its own build in progress. It probes an output before
generating it, probes it again afterwards, and enumerates directories it is
writing into. Those observations describe intra-build state, not the pre-build
state a cache lookup is answering against, and keeping them means the entry can
never match: the probe recorded absence, and by the next build the output is
there.

Two forms of this are handled.

Probes and enumerations of any path this project ever wrote are dropped, along
with every ancestor directory of such a path. The ancestor walk matters because
MSBuild routinely probes an output directory before creating it. "Ever written"
is deliberately broader than the output set, which is filtered to
existing files and would leak transient temporaries and build-created
directories back in. This is project-local, so probes of another project's
outputs still count.

Enumerations of a directory the project writes into cannot simply be dropped --
the directory usually holds real inputs too. Instead its contents are split into
members the project wrote and members it did not. Only the latter go into the
fingerprint; the former are recorded separately so lookup can subtract them from
what it finds on disk. A project that enumerates a staging directory it also
populates then matches whether or not the previous build's outputs are still
there, which is the case that makes clean-to-incremental cycles work.

Members are captured at this layer rather than at fingerprint time because the
self-output set is only known here.

The search pattern is compiled through a shared helper. Patterns come from
whatever the build passed to the enumeration API, and Windows substitutes
wildcards that have no glob equivalent -- DOS_STAR, DOS_QM and DOS_DOT are
spelled `<`, `>` and `"` -- which the glob parser rejects, as it does braces and
a closing bracket. An unparseable pattern falls back to matching every member.
That can only over-invalidate, since the recorded member list becomes a superset;
dropping the observation instead would lose the dependency and risk exactly the
incorrect hit this feature exists to prevent. The helper is shared so that the
capture and validation sides always reach the same conclusion about a pattern.

Everything so far records observations. Nothing yet checks them, and the strong
fingerprint cannot: a probe entry contributes only its identity and an
enumeration entry hashes the member list stored on itself, neither of which
touches the disk. Recomputing the fingerprint from a cached PathSet therefore
always reproduces the fingerprint stored in the selector no matter what the
filesystem now holds. This check is what enforces those entries, and cache
lookup gains nothing by reaching it any earlier than it must.

That is the opposite of how it reads. For file content reads the same check is
purely an optimization, because their bytes are hashed into the fingerprint and
a change is caught by the comparison regardless. Someone could reasonably
conclude the whole thing is redundant and delete it, expecting to pay in cache
hits; they would instead start silently producing incorrect ones. Both call
sites say so.

An existing probe requires the path to still exist and an absent probe requires
it to still be missing -- the latter is the clean-to-dirty case that motivated
the feature, where a file the build only checked for has since appeared.

Directory enumerations re-enumerate, subtract the member names the project wrote
during the populate build, and compare what remains against the recorded
members. Subtracting the self-outputs is what lets the comparison hold whether
or not the previous build's outputs are still on disk.

A recorded member list of null is not an empty directory: it is how the capture
side records a directory it could not enumerate at all, and the fingerprint
already encodes the two differently. So it is validated as absence rather than
by re-enumeration. Re-enumerating would report a miss precisely because nothing
had changed -- the directory is still missing -- making any project that
enumerates a path that does not exist permanently uncacheable, which is an
ordinary consequence of a wildcard over an absent directory. In the other
direction the member comparison treats null and empty alike, so falling through
would let a directory that did not exist and now exists empty validate as
unchanged.

An enumeration that fails mid-way returns no observation at all and forces a
miss, so an IO error can never be mistaken for an empty directory.

Everything the feature needs is in place, so the setting defaults on and the
documented limitation goes away: MSBuildCache no longer assumes the repo is
clean before building. Entries produced before this sit under a different weak
fingerprint and age out through normal eviction, so nothing needs migrating.

Turning it on unconditionally would be wrong on a host that does not report
enumeration patterns. There the capture side records every filtered enumeration
as if it were unfiltered, which costs hits broadly rather than producing
incorrect ones -- but the weak fingerprint would still claim the feature was on.
The setting is therefore forced off on such a host, and forced rather than
merely ignored, so the property stays the single source of truth for behavior,
for logging and for the weak fingerprint instead of reporting a value that
disagrees with what the build did. An explicit opt-out on a capable host is
still honored.

The clamp names the running MSBuild version when it logs, since being told a
feature is off is not actionable without knowing what you are running or that
upgrading would fix it.

The new MSBuild property is added to GlobalPropertiesToIgnore so toggling it
with -p: does not also perturb the global-property hash.

The unit tests exercise the pieces in isolation against synthetic PathSets. What
they cannot show is that a real build, through MSBuild and the sandbox, hits and
misses where it should. Each scenario builds a small generated repo two or three
times, perturbs exactly one thing between builds, and asserts the hit and miss
counts.

The set is chosen so a failure localizes:

* A file added under a target-time dynamic glob misses, and an unchanged source
  tree hits -- the enumeration channel in both directions.
* A marker file appearing where the build only probed for it misses, and a
  marker whose state does not change hits -- the probe channel, and the
  clean-to-dirty case the feature exists for.
* The same marker outside the repo and package roots hits, as does one matched
  by IgnoredInputPatterns, which is what proves the scope and pattern filters
  are load-bearing rather than incidental.
* A project that enumerates a directory it also writes into hits across a
  clean, incremental and clean cycle.
* Editing a source file misses and rebuilding hits, deliberately routed through
  neither new channel, so a failure there points at cache fundamentals rather
  than this feature.
* Three consecutive no-change builds all hit. The third replays over the second
  build's replayed outputs, which catches an observation that only re-validates
  against outputs a real build produced -- something a two-build test passes.

The suite skips with a warning when the MSBuild under test does not report
enumeration patterns, since the feature disables itself there and the scenarios
would assert misses that cannot happen. Failing instead would block every PR
until a suitable MSBuild ships. The warning is raised as a pipeline log issue so
a skip is visible in the build summary rather than buried in a log.

A skip is otherwise indistinguishable from a pass, so the pipeline job that runs
against a bootstrap of dotnet/msbuild main -- the only one carrying the field
today -- requires the capability and fails without it. If the bootstrap layout
or the API changes, that job fails rather than quietly running nothing. The
production Visual Studio jobs keep skipping, and will start running the
scenarios on their own once the field ships.

Builds now tolerate post-completion accesses under the Windows directory: Code
Integrity touching a finished worker otherwise throws from a BuildXL
IO-completion thread and terminates the build. Those paths are outside the repo
and package roots and so are already excluded from observations.

test.ps1 becomes smoke.ps1 now that there is more than one suite.

Allow Windows and Visual Studio telemetry state by location rather than individual filenames, preserving diagnostics without brittle file-specific exemptions.

Copilot-Session: 99f10cf3-31a4-4b84-9afa-bc2a6028bdcf

Keep Application Insights patterns outside repo rooting and cover the assembled Windows and telemetry allowlist against representative positive and negative paths.

Copilot-Session: 99f10cf3-31a4-4b84-9afa-bc2a6028bdcf

MSBuild treats semicolons in /p values as assignment separators. Encode property values at the shared scenario invocation boundary so list-valued settings reach the plugin intact.

Copilot-Session: 99f10cf3-31a4-4b84-9afa-bc2a6028bdcf